↓Skip to main content

Pi for Breakfast

·10 mins
raspberry-pi pi-hole tailscale nomad
Table of Contents

Pi For Breakfast #

The first thing anyone thinks when they get a raspberry-pi:

now what?

Casting about for a problem to my solution I’ve been noticing an egregious amount of ads in my feeds lately - lets fix that now! There are many tutorials about install pi-hole - such as the README itself!

Pi-hole

I wanted to take this ad-blocking server to the next level!

tl;dr here is the tailscale post on this subject: Access a Pi-hole or Raspberry Pi from anywhere. I’d note you need to disable sudo systemctl disable systemd-resolved.service && sudo systemctl stop systemd-resolved.service

This post dives into hardening and monitoring our dns server as a core component to our home network.

Available Anywhere!! #

The one big drawback to self-hosting anything is that unless you have a static IP from your ISP, chances are once you leave home your access to your local network does to.

To resolve this I created a Tailscale account which is free for personal use - perfect in this case. Tailscale allows you to create a VPN (virtual Private Network) between you and your devices. In this case we’re going to create a “local” network with our raspberry-pi being the DNS server.

Once you create an account run tailscale up

Bootstrap Raspberry Pi #

Because this is going to be our DNS server we want a consistent ip address to reach it. By default your router will assign dynamic IPs over DHCP. To assign your Pi a static ip update this file on the boot sector of the SD card:

/etc/netplan/50-cloud-init.yaml

#network:
#    ethernets:
#        eth0:
#            dhcp4: true
#            optional: true
#    version: 2
network:
     ethernets:
        eth0:
          addresses:
            - 192.168.0.26/24
          gateway4: 192.168.0.1
          nameservers:
            addresses: [127.0.0.53]
          optional: true

source: how to install ubuntu on your raspberry pi

192.168.0.26/24 is the static IP I want to assign my pi

192.168.0.1: is the gateway to my local network. You can find this with: ip a | grep enp9s0 (or eth0)

grok a line that looks like this:

inet 192.168.0.24/24 brd 192.168.0.255

network: ethernets: eth0: addresses: - {{ ipv4 address between 192.168.0.1 and 192.168.0.255 }} gateway4: {{ min ip address of the broadcast ip (192.168.0.0) would be your router - so you want .1 }}

Plug-in your Pi and clip an ethernet into your router. Since our Pi is headless we need to find it and shell in.

Test with Nmap #

Nmap is an amazing tool for testing networks - we’re going to use it to find our Pi and make sure its configured properly.

sudo nmap -sn 192.168.0.24/24

and we find an entry like this:

Nmap scan report for 192.168.0.26
Host is up (0.00024s latency).
MAC Address: E4:5F:01:6A:62:71 (Raspberry Pi Trading)

once you’re in you’ll want to do the following:

Install Tailscale #

Follow the official docs here for your respective platform (I installed Ubuntu 20.04.3 LTS).

Once you’ve run tailscale up note the IP address $TAILSCALE_IP.

Harden SSH Access #

1. update your /etc/ssh/sshd_config with your static and tailscale IPs

ListenAddress 192.168.0.26
ListenAddress {{ $TAILSCALE_IP }}

this way if the tailscale proc ever crashes you can still SSH into your pi

I’d also recommend banning PasswordAuthentication` and use your personal SSH key or SSH certificate:

PasswordAuthentication no

You can read more about creating and uploading a personal SSH key here:

run systemctl restart sshd and verify in another tab that you can connect.

What makes this pi so tasty? #

The official Tailscale + Pi-hole docs use the docker-compose stack - while this is great for getting started its hard to know other info such as:

  • is pi-hole up?
  • is it consuming too many resources on my resource-constrained raspberry-pi?
  • is it restarting frequently?
  • where are my logs?

All of these questions can be solved using an orchestrator. I like using Hashicorp’s simple Nomad offering - its easy to get started and great for my simple homelab use case.

Install Docker #

sudo apt install docker.io sudo apt install docker-compose

Follow the post-install instructions here:

2 Docker Post-Install Steps

Install Nomad #

Note we don’t want to use the apt repos here as of this writing they are far behind (0.8.x?).

3 Nomad Deployment Guide

Here is the nomad.hcl for running both the server and client on a raspberry-pi with Docker enabled:

nomad@pi-for-breakfast:~/pi-for-breakfast$ cat /etc/nomad.d/nomad.hcl 
datacenter = "homelab"
data_dir = "/opt/nomad/data"

# tailscale
bind_addr = "$TAILSCALE_IP" <-- note the "double-quotes"

# equivalent to nomad agent -dev
server {
  enabled          = true
  bootstrap_expect = 1
}

client {  
  enabled = true
  servers = ["$TAILSCALE_IP:4647"]
  host_network "tailscale" {
    cidr      = "$TAILSCALE_IP/30"
    interface = "tailscale0"
  }
  node_class = "raspberry-pi"
}

plugin "docker" {
  config {
    endpoint = "unix:///var/run/docker.sock"

    volumes {
      enabled      = true
      selinuxlabel = "z"
    }

    allow_privileged = true
    allow_caps = ["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod", "net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot", "net_admin"]
  }
}

note everything should be owned by the nomad user

run systemctl restart nomad and you should be in business.

Download Pi-hole for Docker #

sudo su - nomad
cd /opt
git clone https://github.com/pi-hole/docker-pi-hole.git

This is how we’ll reference the core configs when mounting them into the Docker container.

Create Pi-hole Nomad job #

One note here that since we are advertising our Nomad server stack using a $TAILSCALE_IP we’ll want to add NOMAD_ADDR to our ~/.bashrc like this:

# Nomad
export NOMAD_ADDR="http://$TAILSCALE_IP:4646"

for both ubuntu and nomad users. Be sure to run source ~/.bashrc afterwards.

job "pi-hole" {
  datacenters = ["homelab"]
  type = "system"
  
  constraint {    
    attribute = "${attr.kernel.name}"
    value     = "linux"  
  }

  constraint {    
    attribute = "${node.class}"    
    value     = "raspberry-pi"  
  }

  group "pi-hole" {
    network {
      mode = "bridge"
      port "dhcp" {
	static       = 67
        to           = 67
        host_network = "tailscale"
      }
      port "dns" {
        static       = 53
        to           = 53
        host_network = "tailscale"
      }
      port "http" {
        static       = 8080
        to           = 80
        host_network = "tailscale"
      }
    }
    task "server" {
      driver = "docker"
      config {        
        image = "pihole/pihole:latest"
        ports = [
          "dns",
          "dhcp",
          "http",
        ]
        volumes  = [
          "/opt/docker-pi-hole/etc-pihole/:/etc/pihole/",
          "/opt/docker-pi-hole/etc-dnsmasq.d/:/etc/dnsmasq.d/",
          "/opt/docker-pi-hole/var-log/pihole.log:/var/log/pihole.log",
        ]
        cap_add = ["net_admin", "setfcap"]
      }
    }
  }
}

gist: Pi-hole on Nomad

Note I bound the http server to 8080 to reserve 80 for a later reverse proxy like haproxy or envoy.

Submit the job to Nomad with:

nomad run pi-hole.nomad

Configure Pi-hole #

reset the password:

docker ps
CONTAINER ID   IMAGE                                      COMMAND      CREATED         STATUS                   PORTS     NAMES
1cdb866fa9a3   pihole/pihole:latest                       "/s6-init"   2 minutes ago   Up 2 minutes (healthy)             server-88047994-f8ef-7b60-18c9-cd52dd631107
e9b653a9e628   gcr.io/google_containers/pause-arm64:3.1   "/pause"     2 minutes ago   Up 2 minutes                       nomad_init_88047994-f8ef-7b60-18c9-cd52dd631107

nomad@pi-for-breakfast:~/nomad-jobs$ docker exec -it 1cdb866fa9a3 bash

root@e9b653a9e628:/# pihole -a -p
Enter New Password (Blank for no password):

???

Profit

Store this in your password manager of choice.

Finally login to your Pi-hole admin panel and setup the configuration like so:

pi-hole configuration: enable Cloudflare DNS + Listen on all interfaces, permit all origins

Note that the last option should not be used on devices which are directly connected to the Internet. This option is safe if your Pi-hole is located within your local network, i.e. protected behind your router, and you have not forwarded port 53 to this device. In virtually all other cases you have to make sure that your Pi-hole is properly firewalled.

✅ Listen on all interfaces, permit all origins

Note that the last option should not be used on devices which are directly connected to the Internet. This option is safe if your Pi-hole is located within your local network, i.e. protected behind your router, and you have not forwarded port 53 to this device. In virtually all other cases you have to make sure that your Pi-hole is properly firewalled.

I’m pasting in this notice to breakdown why this works here - we are using Tailscale to securely connect to our Raspberry-Pi-backed DNS server on our local network. If this were installed on say a VPS like DigitalOcean or Linode we’d have much bigger problems (my remark earlier about using an SSH key or certificate would be mandatory then).

Make sure you enable DNSSEC (Google, Cloudflare, and Pi-hole support it) We’ll verify that it works once we’ve enabled Tailscale DNS…

Tailscale for Days #

Tailscale has a magical feature that enforces DNS servers across all of your devices connected to the Tailscale VPN - this is a great use-case for a DNS server.

1. Login to your Tailscale Account 2. Select DNS 3. Set the nameserver as $TAILSCALE_IP 4. Enable “Override local DNS” - this will revert your /etc/resolv.conf see troubleshooting if your DNS fails to resolve.

Testing #

Dig #

Try forcing DNS lookups via your Pi-hole with:

dig @$TAILSCALE_IP +short google.com

if this hangs make sure you have your pi-hole admin panel configured correctly ^

DNSSEC #

Test that its working with DNSSEC Resolver Test. You’ll want to make sure tailscale status is running on your machine (as well as Pi itself of course) for this to work.

Test Ad-Blocking #

Head over to a site with loads of ads and pull up your network tab - you should see references to NS_LOOKUP failures which means the DNS server (your Pi)_ can’t resolve the address which is what we want!

Verify with Nmap #

Just as we used Nmap to start this project we’ll use Nmap to finish it!

Verify what ports are exposed on your local network:

sudo nmap 192.168.0.26 
Starting Nmap 7.91 ( https://nmap.org ) at 2021-12-11 17:57 CST
Nmap scan report for 192.168.0.26
Host is up (0.00073s latency).
Not shown: 999 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
MAC Address: E4:5F:01:6A:62:71 (Raspberry Pi Trading)

Nmap done: 1 IP address (1 host up) scanned in 0.35 seconds

verify that your Tailscale VPN has the expected ports exposed as well:

nmap 100.92.18.10
Starting Nmap 7.91 ( https://nmap.org ) at 2021-12-11 17:53 CST
Nmap scan report for pi-for-breakfast (100.92.18.10)
Host is up (0.059s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
53/tcp   open  domain
8080/tcp open  http-proxy

Nmap done: 1 IP address (1 host up) scanned in 0.79 seconds

Note that since we’ve bound Nomad to a Tailscale IP that any other services (log shipping, authn, etc) you are running will be available over your Tailscale network. I have plans to exploit this for later projects but might not be ideal for every workload.

Troubleshooting #

One caveat to having Tailscale managing your DNS settings is that during bootstrap (Pi has lost power, Tailscale proc has crashed) we end up in a situation where our devices are depending on our Pi for DNS resolution in addition to itself.

Your very first step when troubleshooting is to dissable “Override local DNS” in your Tailscale account.

Bootstrap Pi-hole Docker images #

failed to setup alloc: pre-run hook "network" failed: failed to create network for alloc: Failed to pull `gcr.io/google_containers/pause-arm64:3.1`: API error (500): Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 100.92.18.10:53: read udp 100.92.18.10:53571->100.92.18.10:53: read: connection refused

temporarilly patch your resolv.conf

cat /etc/resolv.conf
# resolv.conf(5) file generated by tailscale
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

# Tailscale
# nameserver 100.92.18.10

# Google
nameserver 8.8.8.8
nameserver 8.8.4.4

sudo systemctl enable systemd-resolved.service && sudo systemctl restart systemd-resolved.service

dig google.com to verify DNS is now resolving

run nomad run pi-hole.nomad

then enable tailscale “override local DNS”

Run Nomad Client As Root #

failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to initialize table forwarding rules: failed to list iptables chains: running [/usr/sbin/iptables -t filter -S --wait]: exit status 4: Fatal: can't open lock file /run/xtables.lock: Permission denied

update your etc/systemd/system/nomad.service

# Nomad server should be run as the nomad user. Nomad clients
# should be run as root
User=root
Group=nomad

sudo systemctl daemon-reload
sudo systemctl restart nomad

Install CNI Bridge for bridge networking #

failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: failed to find plugin "bridge" in path [/opt/cni/bin]

wget https://github.com/containernetworking/plugins/releases/download/v1.0.1/cni-plugins-linux-arm64-v1.0.1.tgz sudo mkdir -p /opt/cni/bin sudo tar -C /opt/cni/bin -xzf cni-plugins-linux-arm64-v1.0.1.tgz

The benefit of using a bridge network is that resolvd systemD can remain running and act as a fallback if tailscale DNS stops working aka your Raspberry Pi gets unplugged :)

Resources #

1 Install Ubuntu On You Raspberry Pi

2 Docker Post-Install Steps

3 Generating a new SSH key and adding it to the ssh-agent

4 Nomad Deployment Guide

↑